Deep Learning Based Vulnerability Detection: Are We There Yet?

Automated detection of software vulnerabilities is a fundamental problem in security. Existing program analysis techniques either suffer from high false positives or negatives. Recent progress Deep Learning (DL) has resulted surge interest applying DL for automated vulnerability detection. Several recent studies have demonstrated promising results achieving an accuracy up to 95 percent at detecting vulnerabilities. In this paper, we ask, “how well do the state-of-the-art DL-based perform real-world prediction scenario?” To our surprise, find that their performance drops by more than 50 percent. A systematic investigation what causes such precipitous drop reveals existing approaches challenges with training data (e.g., duplication, unrealistic distribution vulnerable classes, etc.) and model choices simple token-based models). As result, these often not learn features related actual cause Instead, they unrelated artifacts dataset specific variable/function names, etc.). Leveraging empirical findings, demonstrate how principled approach collection design, based on realistic settings prediction, can lead better solutions. The resulting tools significantly studied baseline—up 33.57 boost precision 128.38 recall compared best performing literature. Overall, paper elucidates systems’ potential issues draws roadmap future research.